Crate zarrs

source ·
Expand description

A rust library for the Zarr V3 storage format for multidimensional arrays and metadata.

Developed at the Department of Materials Physics, Australian National University, Canberra, Australia.

zarrs is experimental and in limited production use. Use at your own risk! Correctness issues with past versions are detailed here.

A changelog can be found here.

§Getting Started

§Example

let store_path: PathBuf = "/path/to/store".into();
let store: zarrs::storage::ReadableWritableListableStorage =
    Arc::new(zarrs::storage::store::FilesystemStore::new(&store_path)?);

let array_path: &str = "/group/array"; // /path/to/store/group/array
let array = zarrs::array::Array::new(store, array_path)?;

let chunk: ndarray::ArrayD<f32> = array.retrieve_chunk_ndarray(&[1, 0])?;
println!("Chunk [1,0] is:\n{chunk}");

§Implementation Status

Zarr Enhancement Proposals (ZEPs)
Zarr Enhancement ProposalStatusZarrsFeature Flags*
ZEP0001 - Zarr specification version 3AcceptedFull support
ZEP0002 - Sharding codecAcceptedFull supportsharding, crc32c
ZEP0003 - Variable chunkingDraftFull support

* Bolded feature flags are part of the default set of features.

Stores and Storage Adapters
Store/Storage AdapterZEPReadWriteListSyncAsyncFeature Flag
FilesystemStoreZEP0001✓✓✓✓
MemoryStore✓✓✓✓
HTTPStore✓✓http
OpendalStore✓*✓*✓*✓opendal
AsyncOpendalStore✓*✓*✓*✓opendal
AsyncObjectStore✓*✓*✓*✓object_store
ZipStorageAdapter✓✓✓zip

* Support depends on the opendal::BlockingOperator/Operator or object_store store.

Data Types
Codecs
Codec TypeCodecZEPZarrsFeature Flag*
Array to ArraytransposeZEP0001Full supporttranspose
bitroundExperimentalbitround
Array to BytesbytesZEP0001Full support
sharding_indexedZEP0002Full supportsharding
zfpExperimentalzfp
pcodecExperimentalpcodec
Bytes to BytesbloscZEP0001Full supportblosc
gzipZEP0001Full supportgzip
crc32cZEP0002Full supportcrc32c
zstdGitHub PRFull supportzstd
bz2Experimentalbz2

* Bolded feature flags are part of the default set of features.

Experimental codecs are for evaluation purposes only and should not be used in production.
  • They will not be supported by other Zarr V3 implementations until they have been standardised.
  • The "name" of of experimental codecs in array metadata is a link to either:
    • a Zarr ZEP draft or GitHub PR, or
    • the codec documentation in this crate.
Chunk Grids
Chunk GridZEPZarrsFeature Flag
regularZEP0001Full support
rectangularZEP0003Full support
Chunk Key Encodings
Chunk Key EncodingZEPZarrsFeature Flag
defaultZEP0001Full support
v2ZEP0001Full support
Storage Transformers
Storage TransformerZEPZarrsFeature Flag
usage logFull support
performance metricsFull support

§Crate Features

§Default
  • ndarray: ndarray utility functions for Array.
  • Codecs: blosc, gzip, transpose, zstd, sharding, crc32c.
§Non-Default
  • async: an experimental asynchronous API for stores, Array, and Group.
    • The async API is runtime-agnostic. This has some limitations that are detailed in the Array docs.
    • The async API is not as performant as the sync API.
  • Codecs: bitround, bz2, pcodec, zfp, zstd.
  • Stores: http, object_store, opendal, zip.

§Examples

Examples can be run with cargo run --example <EXAMPLE_NAME>.

Add -- --usage-log to see storage API calls during example execution.

Some examples require non-default features, which can be enabled with --features <FEATURES>.

§Sync API

array_write_read, array_write_read_ndarray, sharded_array_write_read, rectangular_array_write_read, zip_array_write_read, http_array_read.

§Async API

async_array_write_read, async_http_array_read.

§zarrs Ecosystem

  • zarrs_tools: Various tools for creating and manipulating Zarr v3 data. Includes zarrs benchmarks.
  • zarrs_ffi: A subset of zarrs exposed as a C API.

§Licence

zarrs is licensed under either of

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Re-exports§

Modules§